

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    
    <title>Data &mdash; Fityk 1.1.1 manual</title>
    <link rel="stylesheet" href="_static/fityk.css" type="text/css" />
    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    '',
        VERSION:     '1.1.1',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true
      };
    </script>
    <script type="text/javascript" src="_static/jquery.js"></script>
    <script type="text/javascript" src="_static/underscore.js"></script>
    <script type="text/javascript" src="_static/doctools.js"></script>
    <link rel="shortcut icon" href="_static/fityk.ico"/>
    <link rel="top" title="Fityk 1.1.1 manual" href="index.html" />
    <link rel="next" title="Models" href="model.html" />
    <link rel="prev" title="Mini-Language" href="lang.html" />
 


  </head>
  <body>
<div style="background-color: white; text-align: center; padding: 10px 10px 10px 10px">
 <a href="http://fityk.nieto.pl/"><img src="_static/fityk-banner.png" alt="" /></a>
</div>

      <div class="sphinxsidebar">
        <div class="sphinxsidebarwrapper">
<h3><a href="fityk-manual.html">Table Of Contents</a></h3>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="intro.html">Introduction</a></li>
<li class="toctree-l1"><a class="reference internal" href="getstarted.html">Getting Started</a></li>
<li class="toctree-l1"><a class="reference internal" href="lang.html">Mini-Language</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="">Data</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#loading-data">Loading Data</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#supported-filetypes">Supported Filetypes</a></li>
<li class="toctree-l3"><a class="reference internal" href="#reading-text-files">Reading Text Files</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#active-and-inactive-points">Active and Inactive Points</a></li>
<li class="toctree-l2"><a class="reference internal" href="#standard-deviation-or-weight">Standard Deviation (or Weight)</a></li>
<li class="toctree-l2"><a class="reference internal" href="#data-point-transformations">Data Point Transformations</a></li>
<li class="toctree-l2"><a class="reference internal" href="#aggregate-functions">Aggregate Functions</a></li>
<li class="toctree-l2"><a class="reference internal" href="#functions-and-variables-in-data-transformation">Functions and Variables in Data Transformation</a></li>
<li class="toctree-l2"><a class="reference internal" href="#working-with-multiple-datasets">Working with Multiple Datasets</a></li>
<li class="toctree-l2"><a class="reference internal" href="#dataset-transformations">Dataset Transformations</a></li>
<li class="toctree-l2"><a class="reference internal" href="#exporting-data">Exporting Data</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="model.html">Models</a></li>
<li class="toctree-l1"><a class="reference internal" href="fit.html">Curve Fitting</a></li>
<li class="toctree-l1"><a class="reference internal" href="ref.html">All the Rest</a></li>
</ul>

  <h3>This Page</h3>
  <ul class="this-page-menu">
    <li><a href="_sources/data.txt"
           rel="nofollow">Show Source</a></li>
  </ul>
<div id="searchbox" style="display: none">
  <h3>Quick search</h3>
    <form class="search" action="search.html" method="get">
      <input type="text" name="q" size="18" />
      <input type="submit" value="Go" />
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
    <p class="searchtip" style="font-size: 90%">
    Enter search terms or a module, class or function name.
    </p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body">
            
  <div class="section" id="data">
<span id="id1"></span><h1>Data</h1>
<div class="section" id="loading-data">
<span id="dataload"></span><h2>Loading Data</h2>
<p>Data files are read using the <a class="reference external" href="http://xylib.sourceforge.net/">xylib library</a>.</p>
<div class="admonition-in-the-gui admonition ">
<p class="first admonition-title">In the GUI</p>
<p class="last">click <img alt="Load Data" class="icon" src="_images/load_data_icon.png" />. If it just works for your files, you may go
straight to <a class="reference internal" href="#activepoints"><em>Active and Inactive Points</em></a>.</p>
</div>
<p>Points are loaded from files using the command:</p>
<div class="highlight-fityk"><div class="highlight"><pre>dataslot &lt; filename[:xcol:ycol:scol:block] [filetype options...]
</pre></div>
</div>
<p>where</p>
<ul class="simple">
<li><em>dataslot</em> should be replaced with <tt class="docutils literal"><span class="pre">&#64;0</span></tt>, unless many datasets
are to be used simultaneously (for details see: <a class="reference internal" href="#multidata"><em>Working with Multiple Datasets</em></a>),</li>
<li><em>xcol</em>, <em>ycol</em>, <em>scol</em> (supported only in text file) are columns
corresponding to x, y and std. dev. of y.
Column 0 means index of the point: 0 for the first point,
1 for the second, etc.</li>
<li><em>block</em> is only supported by formats with multiple blocks of data.</li>
<li><em>filetype</em> usually can be omitted, because in most of the cases
the filetype can be detected; the list of supported filetypes is
at the end of this section</li>
<li><em>options</em> depend on a filetype and usually are omitted</li>
</ul>
<p>If the filename contains blank characters, a semicolon or comma, it
should be put inside single quotation marks (together with colon-separated
indices, if any).</p>
<p>Multiple y columns and/or blocks can be specified, see the examples below:</p>
<div class="highlight-fityk"><div class="highlight"><pre>@0 &lt; foo.vms
@0 &lt; foo.fii text first_line_header
@0 &lt; foo.dat:1:4:: <span class="c"># x,y - 1st and 4th columns</span>
@0 &lt; foo.dat:1:3,4:: <span class="c"># load two dataset (with y in columns 3,4)</span>
@0 &lt; foo.dat:1:3..5:: <span class="c"># load three dataset (with y in columns 3,4,5)</span>
@0 &lt; foo.dat:1:4..6,2:: <span class="c"># load four dataset (y: 4,5,6,2)</span>
@0 &lt; foo.dat:1:2..:: <span class="c"># load 2nd and all the next columns as y</span>
@0 &lt; foo.dat:1:2:3: <span class="c"># read std. dev. of y from 3rd column</span>
@0 &lt; foo.dat:0:1:: <span class="c"># x - 0,1,2,..., y - first column</span>
@0 &lt; <span class="s1">&#39;foo.dat:0:1::&#39;</span> <span class="c"># the same</span>
@0 &lt; foo.raw::::0,1 <span class="c"># load two first blocks of data (as one dataset)</span>
</pre></div>
</div>
<p>Information about loaded data can be obtained with:</p>
<div class="highlight-fityk"><div class="highlight"><pre>info data
</pre></div>
</div>
<div class="section" id="supported-filetypes">
<h3>Supported Filetypes</h3>
<dl class="docutils">
<dt>text</dt>
<dd>ASCII text, multicolumn numeric data.
The details are given in the next section.</dd>
<dt>dbws</dt>
<dd>format used by DBWS (program for Rietveld analysis)
and DMPLOT.</dd>
<dt>cpi</dt>
<dd>Sietronics Sieray CPI format</dd>
<dt>uxd</dt>
<dd>Siemens/Bruker UXD format (powder diffraction data)</dd>
<dt>bruker_raw</dt>
<dd>Simens-Bruker RAW format (version 1,2,3)</dd>
<dt>rigaku_dat</dt>
<dd>Rigaku dat format (powder diffraction data)</dd>
<dt>vamas</dt>
<dd>VAMAS ISO-14976
(only experiment modes: &#8220;SEM&#8221; or &#8220;MAPSV&#8221; or &#8220;MAPSVDP&#8221; and
only &#8220;REGULAR&#8221; scan mode are supported)</dd>
<dt>philips_udf</dt>
<dd>Philips UDF (powder diffraction data)</dd>
<dt>philips_rd</dt>
<dd>Philips RD raw scan format V3 (powder diffraction data)</dd>
<dt>spe</dt>
<dd>Princeton Instruments WinSpec SPE format
(only 1-D data is supported)</dd>
<dt>pdcif</dt>
<dd>CIF for powder diffraction</dd>
</dl>
<p>The full list is available at: <a class="reference external" href="http://xylib.sourceforge.net/">http://xylib.sourceforge.net/</a>.</p>
</div>
<div class="section" id="reading-text-files">
<h3>Reading Text Files</h3>
<p>The <em>xylib</em> library can read TSV or CSV formats (tab or comma separated
values). In fact, the values can be separated by any whitespace character
or by one of ,;: punctations, or by any combination of these.</p>
<p>Empty lines and comments that start with hash (#) are skipped.</p>
<p>Since there is a lot of files in the world that contain numeric data mixed
with text, unless the <em class="xref std std-option">strict</em> option is given
any text that can not be interpreted as a number is regarded a start of
comment (the rest of the line is ignored).</p>
<p>Note that the file is parsed regardless of blocks and columns specified
by the user. The data read from the file are first stored in a table
with <em>m</em> columns and <em>n</em> rows.
If some of the lines have 3 numbers in it, and some have 5 numbers, we can
either discard the lines that have 3 numbers or we can discard the numbers
in 4th and 5th column. Usually the latter is done, but there are exceptions.
The shorter lines are ignored</p>
<ul class="simple">
<li>if it is the last line in the file
(probably the program was terminated while writing the file),</li>
<li>if it contains only one number, but the prior lines had more numbers
(this may be a comment that starts with a number)</li>
<li>if all the (not ignored) prior lines and the next line are longer</li>
</ul>
<p>These rule were introduced to read free-format log files with
textual comments inserted between lines with numeric data.</p>
<p>For now, xylib does not handle well nan&#8217;s and inf&#8217;s in the data.</p>
<p>Data blocks and columns may have names. These names are used to set
a title of the dataset (see <a class="reference internal" href="#multidata"><em>Working with Multiple Datasets</em></a> for details).
If the option <em class="xref std std-option">first_line_header</em> is given and the number of words
in the first line is equal to the number of data columns,
each word is used as a name of corresponding column.
If the number of words is different, the first line is used as a name of the
block.
If the <em class="xref std std-option">last_line_header</em> option is given, the line preceding
the first data line is used to set either column names or the block name.</p>
<p>If the file starts with the &#8220;<tt class="docutils literal"><span class="pre">LAMMPS</span> <span class="pre">(</span></tt>&#8221; string,
the <em class="xref std std-option">last_line_header</em> option is set automatically.
This is very helpful when plotting data from LAMMPS log files.</p>
</div>
</div>
<div class="section" id="active-and-inactive-points">
<span id="activepoints"></span><h2>Active and Inactive Points</h2>
<p>We often have the situation that only a part of the data from a file is
of interest. In Fityk, each point is either <em>active</em> or <em>inactive</em>.
Inactive points are excluded from fitting and all calculations.
A data <a class="reference internal" href="#transform"><em>transformation</em></a>:</p>
<div class="highlight-fityk"><div class="highlight"><pre>A = boolean-condition
</pre></div>
</div>
<p>can be used to change the state of points.</p>
<div class="admonition-in-the-gui admonition ">
<p class="first admonition-title">In the GUI</p>
<p class="last">data points can be activated and disactivated with mouse
in the data-range mode (toolbar: <img alt="Data-Range Mode" class="icon" src="_images/mode_range_icon.png" />).</p>
</div>
</div>
<div class="section" id="standard-deviation-or-weight">
<span id="weights"></span><h2>Standard Deviation (or Weight)</h2>
<p>When fitting data, we assume that only the <em>y</em> coordinate is subject to
statistical errors in measurement. This is a common assumption.
To see how the <em>y</em>&#8216;s standard deviation, <em>σ</em>, influences fitting
(optimization), look at the weighted sum of squared residuals formula
in <a class="reference internal" href="fit.html#nonlinear"><em>Nonlinear Optimization</em></a>.
We can also think about weights of points &#8211; every point has a weight
assigned, that is equal <img class="math" src="_images/math/d953af3188b3c252a5220e76255e375ee0afaf07.png" alt="w_i=1/\sigma_i^2" style="vertical-align: -5px"/>.</p>
<p>Standard deviation of points can be
<a class="reference internal" href="#dataload"><em>read from file</em></a> together with the <em>x</em> and <em>y</em>
coordinates. Otherwise, it is set either to max(<em>y</em><sup>1/2</sup>, 1)
or to 1, depending on the <em class="xref std std-option">default_sigma</em> option.
Setting std. dev. as a square root of the value is common
and has theoretical ground when <em>y</em> is the number of independent events.
You can always change the standard deviation, e.g. make it equal for every
point with the command: <tt class="docutils literal"><span class="pre">S=1</span></tt>.
See <a class="reference internal" href="#transform"><em>Data Point Transformations</em></a> for details.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">It is often the case that user is not sure what standard deviation
should be assumed, but it is her responsibility to pick something.</p>
</div>
</div>
<div class="section" id="data-point-transformations">
<span id="transform"></span><h2>Data Point Transformations</h2>
<p>Every data point has four properties: <em>x</em> coordinate, <em>y</em> coordinate,
standard deviation of <em>y</em> and active/inactive flag.
These properties can be changed using symbols <tt class="docutils literal"><span class="pre">X</span></tt>, <tt class="docutils literal"><span class="pre">Y</span></tt>, <tt class="docutils literal"><span class="pre">S</span></tt> and <tt class="docutils literal"><span class="pre">A</span></tt>,
respectively. It is possible to either change a single point or apply
a transformation to all points. For example:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">Y[3]=1.2</span></tt> assigns the <em>y</em> coordinate of the 4th point (0 is first),</li>
<li><tt class="docutils literal"><span class="pre">Y</span> <span class="pre">=</span> <span class="pre">-y</span></tt> changes the sign of the <em>y</em> coordinate for all points.</li>
</ul>
<p>On the left side of the equality sign you can have one of symbols <tt class="docutils literal"><span class="pre">X</span></tt>, <tt class="docutils literal"><span class="pre">Y</span></tt>,
<tt class="docutils literal"><span class="pre">S</span></tt>, <tt class="docutils literal"><span class="pre">A</span></tt>, possibly with the index in brackets. The symbols on the left
side are case insensitive.</p>
<p>The right hand side is a mathematical expression that can have special
variables:</p>
<ul class="simple">
<li>lower case letters <tt class="docutils literal"><span class="pre">x</span></tt>, <tt class="docutils literal"><span class="pre">y</span></tt>, <tt class="docutils literal"><span class="pre">s</span></tt>, <tt class="docutils literal"><span class="pre">a</span></tt> represent properties of data
points before transformation,</li>
<li>upper case <tt class="docutils literal"><span class="pre">X</span></tt>, <tt class="docutils literal"><span class="pre">Y</span></tt>, <tt class="docutils literal"><span class="pre">S</span></tt>, <tt class="docutils literal"><span class="pre">A</span></tt> stand for the same properties
after transformation,</li>
<li><tt class="docutils literal"><span class="pre">M</span></tt> stands for the number of points.</li>
<li><tt class="docutils literal"><span class="pre">n</span></tt> stands for the index of currently transformed point,
e.g., <tt class="docutils literal"><span class="pre">Y=y[M-n-1]</span></tt> means that <em>n</em>-th point (<em>n</em>=0, 1, ... M-1)
is assigned <em>y</em> value of the <em>n</em>-th point from the end.</li>
</ul>
<p>Before the transformation a new array of points is created as a copy of the
old array.
Operations are applied sequentially from the first point to the last one,
so while <tt class="docutils literal"><span class="pre">Y[n+1]</span></tt> and <tt class="docutils literal"><span class="pre">y[n+1]</span></tt> have always the same value,
<tt class="docutils literal"><span class="pre">Y[n-1]</span></tt> and <tt class="docutils literal"><span class="pre">y[n-1]</span></tt> may differ. For example, the two commands:</p>
<div class="highlight-fityk"><div class="highlight"><pre>Y = y[n] + y[n-1]
Y = y[n] + Y[n-1]
</pre></div>
</div>
<p>differ. The first one adds to each point the value of the previous point.
The second one adds the value of the previous point <em>after</em> transformation,
so effectively it adds the sum of all previous points.
The index <tt class="docutils literal"><span class="pre">[n]</span></tt> could be omitted (<tt class="docutils literal"><span class="pre">Y</span> <span class="pre">=</span> <span class="pre">y</span> <span class="pre">+</span> <span class="pre">y[n-1]</span></tt>).
The value of undefined points, like <tt class="docutils literal"><span class="pre">y[-1]</span></tt> and <tt class="docutils literal"><span class="pre">Y[-1]</span></tt>,
is explained later in this section.</p>
<p>Expressions can contain:</p>
<ul class="simple">
<li>real numbers in normal or scientific format (e.g. <tt class="docutils literal"><span class="pre">1.23e5</span></tt>),</li>
<li>constants <tt class="docutils literal"><span class="pre">pi</span></tt>, <tt class="docutils literal"><span class="pre">true</span></tt> (1), <tt class="docutils literal"><span class="pre">false</span></tt> (0)</li>
<li>binary operators: <tt class="docutils literal"><span class="pre">+</span></tt>, <tt class="docutils literal"><span class="pre">-</span></tt>, <tt class="docutils literal"><span class="pre">*</span></tt>, <tt class="docutils literal"><span class="pre">/</span></tt>, <tt class="docutils literal"><span class="pre">^</span></tt>,</li>
<li>boolean operators: <tt class="docutils literal"><span class="pre">and</span></tt>, <tt class="docutils literal"><span class="pre">or</span></tt>, <tt class="docutils literal"><span class="pre">not</span></tt>,</li>
<li>comparisions: <tt class="docutils literal"><span class="pre">&gt;</span></tt>, <tt class="docutils literal"><span class="pre">&gt;=</span></tt>, <tt class="docutils literal"><span class="pre">&lt;</span></tt>, <tt class="docutils literal"><span class="pre">&lt;=</span></tt>, <tt class="docutils literal"><span class="pre">==</span></tt>, <tt class="docutils literal"><span class="pre">!=</span></tt>.</li>
<li>one argument functions:<ul>
<li><tt class="docutils literal"><span class="pre">sqrt</span></tt></li>
<li><tt class="docutils literal"><span class="pre">exp</span></tt></li>
<li><tt class="docutils literal"><span class="pre">log10</span></tt></li>
<li><tt class="docutils literal"><span class="pre">ln</span></tt></li>
<li><tt class="docutils literal"><span class="pre">sin</span></tt></li>
<li><tt class="docutils literal"><span class="pre">cos</span></tt></li>
<li><tt class="docutils literal"><span class="pre">tan</span></tt></li>
<li><tt class="docutils literal"><span class="pre">sinh</span></tt></li>
<li><tt class="docutils literal"><span class="pre">cosh</span></tt></li>
<li><tt class="docutils literal"><span class="pre">tanh</span></tt></li>
<li><tt class="docutils literal"><span class="pre">atan</span></tt></li>
<li><tt class="docutils literal"><span class="pre">asin</span></tt></li>
<li><tt class="docutils literal"><span class="pre">acos</span></tt></li>
<li><tt class="docutils literal"><span class="pre">erf</span></tt></li>
<li><tt class="docutils literal"><span class="pre">erfc</span></tt></li>
<li><tt class="docutils literal"><span class="pre">gamma</span></tt></li>
<li><tt class="docutils literal"><span class="pre">lgamma</span></tt> (=ln(|<tt class="docutils literal"><span class="pre">gamma()</span></tt>|))</li>
<li><tt class="docutils literal"><span class="pre">abs</span></tt></li>
<li><tt class="docutils literal"><span class="pre">round</span></tt> (rounds to the nearest integer)</li>
</ul>
</li>
<li>two argument functions:<ul>
<li><tt class="docutils literal"><span class="pre">mod</span></tt> (modulo)</li>
<li><tt class="docutils literal"><span class="pre">min2</span></tt></li>
<li><tt class="docutils literal"><span class="pre">max2</span></tt> (<tt class="docutils literal"><span class="pre">max2(3,5)</span></tt> gives 5),</li>
<li><tt class="docutils literal"><span class="pre">randuniform(a,</span> <span class="pre">b)</span></tt> (random number from interval (a, b)),</li>
<li><tt class="docutils literal"><span class="pre">randnormal(mu,</span> <span class="pre">sigma)</span></tt> (random number from normal distribution),</li>
<li><tt class="docutils literal"><span class="pre">voigt(a,</span> <span class="pre">b)</span></tt>
= <img class="math" src="_images/math/d3e7d7cd926970086d34700ac3f979eac398132a.png" alt="\frac{b}{\pi} \int_{-\infty}^{+\infty} \frac{\exp(-t^2)}{b^2+(a-t)^2} dt" style="vertical-align: -9px"/></li>
</ul>
</li>
<li>ternary <tt class="docutils literal"><span class="pre">?:</span></tt> operator: <tt class="docutils literal"><span class="pre">condition</span> <span class="pre">?</span>&nbsp; <span class="pre">expression1</span> <span class="pre">:</span> <span class="pre">expression2</span></tt>,
which returns <em>expression1</em> if condition is true and <em>expression2</em> otherwise.</li>
</ul>
<p>A few examples.</p>
<ul>
<li><p class="first">The <em>x</em> scale of diffraction pattern can be changed from 2<em>θ</em> to <em>Q</em>:</p>
<div class="highlight-fityk"><div class="highlight"><pre>X = 4*pi * sin(x/2*pi/180) / 1.54051 <span class="c"># Cu 2θ -&gt; Q</span>
</pre></div>
</div>
</li>
<li><p class="first">Negative <em>y</em> values can be zeroed:</p>
<div class="highlight-fityk"><div class="highlight"><pre>Y = max2(y, 0)
</pre></div>
</div>
</li>
<li><p class="first">All standard deviations can be set to 1:</p>
<div class="highlight-fityk"><div class="highlight"><pre>S = 1
</pre></div>
</div>
</li>
<li><p class="first">It is possible to select active range of data:</p>
<div class="highlight-fityk"><div class="highlight"><pre>A = x &gt; 40 and x &lt; 60 <span class="c"># select range (40, 60)</span>
</pre></div>
</div>
</li>
</ul>
<p>All operations are performed on <strong>real numbers</strong>.
Two numbers that differ less than <em>ε</em>
(the value of <em>ε</em> is set by the <a class="reference internal" href="ref.html#epsilon"><em>option epsilon</em></a>)
are considered equal.</p>
<p>Points can be created or deleted by changing the value of <tt class="docutils literal"><span class="pre">M</span></tt>.
For example, the following commands:</p>
<div class="highlight-fityk"><div class="highlight"><pre>M=500; x=n/100; y=sin(x)
</pre></div>
</div>
<p>create 500 points and generate a sinusoid.</p>
<p>Points are kept sorted according to their <em>x</em> coordinate.
The sorting is performed after each transformation.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Changing the <em>x</em> coordinate may change the order
and indices of points.</p>
</div>
<p>Indices, like all other values, are computed in the real number domain.
If the index is not integer (it is compared using <em>ε</em> to the rounded value):</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">x</span></tt>, <tt class="docutils literal"><span class="pre">y</span></tt>, <tt class="docutils literal"><span class="pre">s</span></tt>, <tt class="docutils literal"><span class="pre">a</span></tt> are interpolated linearly.
For example, <tt class="docutils literal"><span class="pre">y[2.5]</span></tt> is equal to <tt class="docutils literal"><span class="pre">(y[2]+[3])/2</span></tt>.
If the index is less than 0 or larger than M-1, the value for the first
or the last point, respectively, is returned.</li>
<li>For <tt class="docutils literal"><span class="pre">X</span></tt>, <tt class="docutils literal"><span class="pre">Y</span></tt>, <tt class="docutils literal"><span class="pre">S</span></tt>, <tt class="docutils literal"><span class="pre">A</span></tt> the index is rounded to integer.
If the index is less than 0 or larger than M-1, 0 is returned.</li>
</ul>
<p>Transformations separated by commas (<tt class="docutils literal"><span class="pre">,</span></tt>) form a sequance of transformations.
During the sequance, the vectors <tt class="docutils literal"><span class="pre">x</span></tt>, <tt class="docutils literal"><span class="pre">y</span></tt>, <tt class="docutils literal"><span class="pre">s</span></tt> and <tt class="docutils literal"><span class="pre">a</span></tt> that contain
old values are not changed. This makes possible to swap the axes:</p>
<div class="highlight-fityk"><div class="highlight"><pre>X=y, Y=x
</pre></div>
</div>
<p>The special <tt class="docutils literal"><span class="pre">index(arg)</span></tt> function returns the index of point that has
<em>x</em> equal <em>arg</em>, or, if there is no such point, the linear interpolation
of two neighbouring indices. This enables equilibrating the step of data
(with interpolation of <em>y</em> and <em>σ</em>):</p>
<div class="highlight-fityk"><div class="highlight"><pre>X = x[0] + n * (x[M-1]-x[0]) / (M-1), Y = y[index(X)], S = s[index(X)]
</pre></div>
</div>
<p>It is possible to delete points for which given condition is true,
using expression <tt class="docutils literal"><span class="pre">delete(condition)</span></tt>:</p>
<div class="highlight-fityk"><div class="highlight"><pre>delete(not a) <span class="c"># delete inactive points</span>

<span class="c"># reduce twice the number of points, averaging x and adding y</span>
x = (x[n]+x[n+1])/2
y = y[n]+y[n+1]
delete(mod(n,2) == 1)
</pre></div>
</div>
<p>If you have more than one dataset, you may need to specify to which
dataset the transformation applies. See <a class="reference internal" href="#multidata"><em>Working with Multiple Datasets</em></a> for details.</p>
<p>The value of a data expression can be shown using the <tt class="docutils literal"><span class="pre">print</span></tt> command.
The precision of printed numbers is governed by the
<a class="reference internal" href="ref.html#numeric-format"><em>numeric_format</em></a> option.</p>
<div class="highlight-fityk"><div class="highlight"><pre>print M <span class="c"># the number of points</span>
print y[index(20)] <span class="c"># value of y for x=20</span>
</pre></div>
</div>
</div>
<div class="section" id="aggregate-functions">
<h2>Aggregate Functions</h2>
<p>Aggregate functions have syntax:</p>
<div class="highlight-fityk"><div class="highlight"><pre>aggregate(expression [if condition])
</pre></div>
</div>
<p>and return a single value, calculated from values of all points
for which the given condition is true. If the condition is omitted, all points
in the dataset are taken into account.</p>
<p>The following aggregate functions are recognized:</p>
<ul>
<li><p class="first"><tt class="docutils literal"><span class="pre">min()</span></tt> &#8212; the smallest value,</p>
</li>
<li><p class="first"><tt class="docutils literal"><span class="pre">max()</span></tt> &#8212; the largest value,</p>
</li>
<li><dl class="first docutils">
<dt><tt class="docutils literal"><span class="pre">argmin()</span></tt> &#8212; (stands for the argument of the minimum)</dt>
<dd><p class="first last">the x value of the point for which the expression
in brackets has the smallest value,</p>
</dd>
</dl>
</li>
<li><dl class="first docutils">
<dt><tt class="docutils literal"><span class="pre">argmax()</span></tt> &#8212; the x value of the point for which the expression</dt>
<dd><p class="first last">in brackets has the largest value,</p>
</dd>
</dl>
</li>
<li><p class="first"><tt class="docutils literal"><span class="pre">sum()</span></tt> &#8212; the sum,</p>
</li>
<li><p class="first"><tt class="docutils literal"><span class="pre">count()</span></tt> &#8212; the number of points for which the expression is true,</p>
</li>
<li><p class="first"><tt class="docutils literal"><span class="pre">avg()</span></tt> &#8212; the arithmetic mean,</p>
</li>
<li><p class="first"><tt class="docutils literal"><span class="pre">stddev()</span></tt> &#8212; the standard deviation,</p>
</li>
<li><p class="first"><tt class="docutils literal"><span class="pre">darea()</span></tt> &#8212; a function used to normalize the area (see the example below).
It returns the sum of
<em>expression</em>*(<em>x</em>[<em>n</em>+1]-<em>x</em>[<em>n</em>-1])/2.
In particular, <tt class="docutils literal"><span class="pre">darea(y)</span></tt> returns the interpolated area under
data points.</p>
</li>
</ul>
<p>Examples:</p>
<div class="highlight-fityk"><div class="highlight"><pre>p avg(y) <span class="c"># print the average y value</span>
p max(y) <span class="c"># the largest y value</span>
p argmax(y) <span class="c"># the position of data maximum</span>
p max(y if x &gt; 40 and x &lt; 60)   <span class="c"># the largest y value for x in (40, 60)</span>
p max(y if a) <span class="c"># the largest y value in the active range</span>
p min(x if y &gt; 0.1)] <span class="c"># x of the first point with y &gt; 0.1</span>
p count(y&gt;100) <span class="c"># the number of points that have y above 100</span>
p count(y&gt;avg(y)) <span class="c"># aggregate functions can be nested</span>
p y[min(n if y &gt; 100)] <span class="c"># the first (from the left) value of y above 100</span>

<span class="c"># take the first 2000 points, average them and subtract as background</span>
Y = y - avg(y if n&lt;2000)

Y = y / darea(y) <span class="c"># normalize data area</span>

<span class="c"># make active only the points on the left from the first</span>
<span class="c"># point with y &gt; 0.1</span>
a = x &lt; min(x if y &gt; 0.1)]
</pre></div>
</div>
</div>
<div class="section" id="functions-and-variables-in-data-transformation">
<span id="funcindt"></span><h2>Functions and Variables in Data Transformation</h2>
<p>You may postpone reading this section and read about the <a class="reference internal" href="model.html#model"><em>Models</em></a> first.</p>
<p>Variables ($foo) and functions (%bar) can be used in data expressions:</p>
<div class="highlight-fityk"><div class="highlight"><pre>Y = y / $foo  <span class="c"># divides all y&#39;s by $foo</span>
Y = y - %f(x) <span class="c"># subtracts function %f from data</span>
Y = y - @0.F(x) <span class="c"># subtracts all functions in F</span>

<span class="c"># print the abscissa value of the maximum of the model</span>
<span class="c"># (only the values in points are considered,</span>
<span class="c">#  so it&#39;s not exactly the model&#39;s maximum)</span>
print argmax(F(x))

<span class="c"># print the maximum of the sum of two functions</span>
print max(%_1(x) + %_2(x))

<span class="c"># Fit constant x-correction (i.e. fit the instrumental zero error), ...</span>
Z = Constant(~0)
fit
X = x + Z(x)        <span class="c"># ... correct the data</span>
Z = 0               <span class="c"># ... and remove the correction from the model.</span>
</pre></div>
</div>
<div class="admonition-in-the-gui admonition ">
<p class="first admonition-title">In the GUI</p>
<p>in the <em>Baseline Mode</em> (<img alt="Baseline Mode" class="icon" src="_images/mode_bg_icon1.png" />),
functions <tt class="docutils literal"><span class="pre">Spline</span></tt> and <tt class="docutils literal"><span class="pre">Polyline</span></tt>
are used to subtract manually selected background.
Clicking <img alt="Strip Background" class="icon" src="_images/strip_bg_icon1.png" /> results in a command like this:</p>
<div class="highlight-fityk"><div class="highlight"><pre>%bg0 = Spline(14.2979,62.1253, 39.5695,35.0676, 148.553,49.9493)
Y = y - %bg0(x)
</pre></div>
</div>
<p>Clicking the same button again undoes the subtraction:</p>
<div class="highlight-fityk"><div class="highlight"><pre>Y = y + %bg0(x)
</pre></div>
</div>
<p class="last">The function edited in the <em>Baseline Mode</em> is always named <tt class="docutils literal"><span class="pre">%bgX</span></tt>,
where <em>X</em> is the index of the dataset.</p>
</div>
<p>Values of the function parameters (e.g. <tt class="docutils literal"><span class="pre">%fun.a0</span></tt>) and pseudo-parameters
<tt class="docutils literal"><span class="pre">Center</span></tt>, <tt class="docutils literal"><span class="pre">Height</span></tt>, <tt class="docutils literal"><span class="pre">FWHM</span></tt> and <tt class="docutils literal"><span class="pre">Area</span></tt> (e.g. <tt class="docutils literal"><span class="pre">%fun.Area</span></tt>)
can also be used.
Pseudo-parameters are supported only by functions, which know
how to calculate these properties.</p>
<p>It is also possible to calculate some properties of %functions:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">%f.numarea(x1,</span> <span class="pre">x2,</span> <span class="pre">n)</span></tt> gives area integrated numerically
from <em>x1</em> to <em>x2</em> using trapezoidal rule with <em>n</em> equal steps.</li>
<li><tt class="docutils literal"><span class="pre">%f.findx(x1,</span> <span class="pre">x2,</span> <span class="pre">y)</span></tt> finds <em>x</em> in interval (<em>x1</em>, <em>x2</em>) such that
%f(<em>x</em>)=<em>y</em> using bisection method combined with Newton-Raphson method.
It is a requirement that %f(<em>x1</em>) &lt; <em>y</em> &lt; %f(<em>x2</em>).</li>
<li><tt class="docutils literal"><span class="pre">%f.extremum(x1,</span> <span class="pre">x2)</span></tt> finds <em>x</em> in interval (<em>x1</em>, <em>x2</em>)
such that %f&#8217;(<em>x</em>)=0 using bisection method.
It is a requirement that %f&#8217;(<em>x1</em>) and %f&#8217;(<em>x2</em>) have different signs.</li>
</ul>
<p>A few examples:</p>
<div class="highlight-fityk"><div class="highlight"><pre>print %fun.numarea(, 0, 100, 10000) <span class="c"># shows area of function %fun</span>
print %_1(%_1.extremum(40, 50)) <span class="c"># shows extremum value</span>

<span class="c"># calculate FWHM numerically, value 50 can be tuned</span>
$c = {%f.Center}
p %f.findx($c, $c+50, %f.Height/2) - %f.findx($c, $c-50, %f.Height/2)
p %f.FWHM <span class="c"># should give almost the same.</span>
</pre></div>
</div>
</div>
<div class="section" id="working-with-multiple-datasets">
<span id="multidata"></span><h2>Working with Multiple Datasets</h2>
<p>Let us call a set of data that usually comes from one file &#8211;
a <em class="dfn">dataset</em>. It is possible to work simultaneously with multiple datasets.
Datasets have numbers and are referenced by <tt class="docutils literal"><span class="pre">&#64;</span></tt> with the number,
(e.g. <tt class="docutils literal"><span class="pre">&#64;3</span></tt>).
The user can specify which dataset the command should be applied to:</p>
<div class="highlight-fityk"><div class="highlight"><pre>@0: M=500    <span class="c"># change the number of points in the first dataset</span>
@1 @2: M=500 <span class="c"># the same command applied to two datasets</span>
@*: M=500    <span class="c"># and the same applied to all datasets</span>
</pre></div>
</div>
<p>If the dataset is not specified, the command applies to the default dataset,
which is initially &#64;0. The <tt class="docutils literal"><span class="pre">use</span></tt> command changes the default dataset:</p>
<div class="highlight-fityk"><div class="highlight"><pre>use @2 <span class="c"># set @2 as default</span>
</pre></div>
</div>
<p>To load dataset from file, use one of the commands:</p>
<div class="highlight-fityk"><div class="highlight"><pre>@n &lt; filename:xcol:ycol:scol:block filetype options...

@+ &lt; filename:xcol:ycol:scol:block filetype options...
</pre></div>
</div>
<p>The first one uses existing data slot and the second one creates
a new slot.  Using <tt class="docutils literal"><span class="pre">&#64;+</span></tt> increases the number of datasets,
and the command <tt class="docutils literal"><span class="pre">delete</span> <span class="pre">&#64;n</span></tt> decreases it.</p>
<p>The dataset can be duplicated (<tt class="docutils literal"><span class="pre">&#64;+</span> <span class="pre">=</span> <span class="pre">&#64;n</span></tt>) or transformed,
more on this in <a class="reference internal" href="#datasettr"><em>the next section</em></a>.</p>
<p>Each dataset has a separate <a class="reference internal" href="model.html#model"><em>model</em></a>,
that can be fitted to the data. This is explained in the next chapter.</p>
<p>Each dataset also has a title (it does not have to be unique, however).
When loading file, a title is automatically created:</p>
<ul class="simple">
<li>if there is a name associated with the column <em>ycol</em>, the title
is based on it;</li>
<li>otherwise, if there is a name associated with the data block read from file,
the title is set to this name;</li>
<li>otherwise, the title is based on the filename</li>
</ul>
<p>Titles can be changed using the command:</p>
<div class="highlight-fityk"><div class="highlight"><pre>@n: title = <span class="s1">&#39;new-title&#39;</span>
</pre></div>
</div>
<p>To print the title of the dataset, type <tt class="docutils literal"><span class="pre">&#64;n:</span> <span class="pre">info</span> <span class="pre">title</span></tt>.</p>
</div>
<div class="section" id="dataset-transformations">
<span id="datasettr"></span><h2>Dataset Transformations</h2>
<p>There are a few transformations defined for a whole dataset
or for two datasets. The syntax is <tt class="docutils literal"><span class="pre">&#64;n</span> <span class="pre">=</span> <span class="pre">...</span></tt> or <tt class="docutils literal"><span class="pre">&#64;+</span> <span class="pre">=</span> <span class="pre">...</span></tt>.
The the right hand side expression supports the following operations:</p>
<dl class="docutils">
<dt><tt class="docutils literal"><span class="pre">-&#64;n</span></tt></dt>
<dd>negation of all <em>y</em> values,</dd>
<dt><tt class="docutils literal"><span class="pre">d</span> <span class="pre">*</span> <span class="pre">&#64;n</span></tt></dt>
<dd>(e.g. <tt class="docutils literal"><span class="pre">0.4*&#64;0</span></tt>) <em>y</em> values are multiplied by <em>d</em>,</dd>
<dt><tt class="docutils literal"><span class="pre">&#64;n</span> <span class="pre">+</span> <span class="pre">&#64;m</span></tt></dt>
<dd>returns <tt class="docutils literal"><span class="pre">&#64;n</span></tt> with added <em>y</em> values from interpolated <tt class="docutils literal"><span class="pre">&#64;m</span></tt>,</dd>
<dt><tt class="docutils literal"><span class="pre">&#64;n</span> <span class="pre">-</span> <span class="pre">&#64;m</span></tt></dt>
<dd>returns <tt class="docutils literal"><span class="pre">&#64;n</span></tt> with subtracted <em>y</em> values from interpolated <tt class="docutils literal"><span class="pre">&#64;m</span></tt>,</dd>
<dt><tt class="docutils literal"><span class="pre">&#64;n</span> <span class="pre">and</span> <span class="pre">&#64;m</span></tt></dt>
<dd>returns points from both datasets (re-sorted),</dd>
</dl>
<p>and functions:</p>
<dl class="docutils">
<dt><tt class="docutils literal"><span class="pre">sum_same_x(&#64;n)</span></tt></dt>
<dd>Merges points which have distance in <em>x</em> is smaller than
<a class="reference internal" href="ref.html#epsilon"><em>epsilon</em></a>.
<em>x</em> of the merged point is the average,
and <em>y</em> and <em>σ</em> are sums of components.</dd>
<dt><tt class="docutils literal"><span class="pre">avg_same_x(&#64;n)</span></tt></dt>
<dd>The same as <tt class="docutils literal"><span class="pre">sum_same_x</span></tt>, but <em>y</em> and <em>σ</em> are set as the average
of components.</dd>
<dt><tt class="docutils literal"><span class="pre">shirley_bg(&#64;n)</span></tt></dt>
<dd>Calculates Shirley background
(useful in X-ray photoelectron spectroscopy).</dd>
</dl>
<p>Examples:</p>
<div class="highlight-fityk"><div class="highlight"><pre>@+ = @0 <span class="c"># duplicate the dataset</span>
@+ = @0 and @1 <span class="c"># create a new dataset from @0 and @1</span>
@0 = @0 - shirley_bg(@0) <span class="c"># remove Shirley background</span>
@0 = @0 - @1 <span class="c"># subtract @1 from @0</span>
@0 = @0 - 0.28*@1 <span class="c"># subtract scaled dataset @1 from @0</span>
</pre></div>
</div>
</div>
<div class="section" id="exporting-data">
<span id="dexport"></span><h2>Exporting Data</h2>
<p>Command:</p>
<div class="highlight-fityk"><div class="highlight"><pre>print all: expression, ... &gt; file.tsv
</pre></div>
</div>
<p>can export data to an ASCII TSV (tab separated values) file.</p>
<div class="admonition-in-the-gui admonition ">
<p class="first admonition-title">In the GUI</p>
<p class="last"><em class="menuselection">Data ‣ Export</em></p>
</div>
<p>To export data in a 3-column (x, y and standard deviation) format, use:</p>
<div class="highlight-fityk"><div class="highlight"><pre>print all: x, y, s &gt; file.tsv
</pre></div>
</div>
<p>Any expressions can be printed out:</p>
<div class="highlight-fityk"><div class="highlight"><pre>p all: n+1, x, y, F(x), y-F(x), %foo(x), sin(pi*x)+y^2 &gt; file.tsv
</pre></div>
</div>
<p>It is possible to select which points are to be printed by replacing <tt class="docutils literal"><span class="pre">all</span></tt>
with <tt class="docutils literal"><span class="pre">if</span></tt> followed by a condition:</p>
<div class="highlight-fityk"><div class="highlight"><pre>print if a: x, y <span class="c"># only active points are printed</span>
print if x &gt; 30 and x &lt; 40: x, y <span class="c"># only points in (30,40)</span>
</pre></div>
</div>
<p>The option <a class="reference internal" href="ref.html#numeric-format"><em>numeric_format</em></a>
controls the format and precision of all numbers.</p>
</div>
</div>


          </div>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="model.html" title="Models"
             accesskey="N">next</a></li>
        <li class="right" >
          <a href="lang.html" title="Mini-Language"
             accesskey="P">previous</a> &nbsp; </li>
  <li><a href="http://fityk.nieto.pl/">Home</a>&nbsp;&nbsp;&nbsp;</li>
  <li><a href="https://github.com/wojdyr/fityk/wiki">Wiki</a>&nbsp;&nbsp;&nbsp;</li>
  
        <li><a href="fityk-manual.html">Manual</a> </li>
 
      </ul>
    </div>
    <div class="footer">
      Last updated on 2011-09-28.
      Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.0.7.
    </div>
  </body>
</html>